Method and device for recognizing motion

ABSTRACT

A method and device for recognizing a motion of an object, the method including receiving event signals from a vision sensor configured to sense the motion, storing, in an event map, first time information indicating a time at which intensity of light corresponding to the event signals changes; generating an image based on second time information corresponding to a predetermined time range among the first time information, and recognizing the motion of the object based on the image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. patentapplication Ser. No. 14/986,851, filed on Jan. 4, 2016, which claimspriority from Korean Patent Application No. 10-2015-0107962, filed onJul. 30, 2015, in the Korean Intellectual Property Office, thedisclosures of which are incorporated herein in their entirety byreference.

BACKGROUND 1. Field

Methods and devices consistent with exemplary embodiments relate torecognizing a motion.

2. Description of the Related Art

In an image-based method of recognizing a motion based on image dataoutput from a frame-based vision sensor, a motion of an objectrepresented based on a plurality of images is recognized. Each of theimages includes static state information of the object such that a largenumber of images is required for recognizing the motion of the objectbased on the images. Therefore, in the image-based method of recognizinga motion based on image data output from a frame-based vision sensor, alarge amount of operation and time are required according to a highcomplexity for performing an operation of the plurality of images suchthat an optimal performance of recognizing the motion may not beguaranteed.

SUMMARY

Exemplary embodiments may address at least the above problems and/ordisadvantages and other disadvantages not described above. Also, theexemplary embodiments are not required to overcome the disadvantagesdescribed above, and an exemplary embodiment may not overcome any of theproblems described above.

According to an aspect of an exemplary embodiment, there is providedmethod of recognizing a motion of an object, the method includingreceiving event signals from a vision sensor configured to sense themotion, storing, in an event map, first time information indicating atime at which intensity of light corresponding to the event signalschanges, generating an image based on second time informationcorresponding to a predetermined time range among the first timeinformation, and recognizing the motion of the object based on theimage.

The generating the image may include generating a first image includingtime information corresponding to a first time range among the firsttime information and generating a second image comprising timeinformation corresponding to a second time range among the first timeinformation, the second time range being different from the first timerange.

The recognizing the motion may include determining context informationbased on the first image and recognizing the motion of the objectincluded in the second image based on the context information.

The first time range may be wider than the second time range.

One end of the first time range and one end of the second time range maycorrespond to an identical point in time.

The recognizing of the motion of the object may include recognizing themotion of the object from the image based on a neural network.

The event map may include a two-dimensional (2D) map corresponding tothe vision sensor and include time information in which most recentlygenerated changes in intensity of light correspond to the event signals.

The event map may include a three-dimensional (3D) map generated byadding a time axis to the 2D map corresponding to the vision sensor andinclude a time information history.

The vision sensor may include an event-based vision sensor configured togenerate at least one event signal in response to an event in whichlight received from the object is asynchronously changed.

The first time information is information of a time at which the eventsignals are received from the vision sensor or information of a time atwhich the event signals are generated by the vision sensor.

According to an aspect of another exemplary embodiment, there isprovided a non-transitory computer-readable storage medium storing aprogram that is executable by a computer to perform the method.

According to an aspect of another exemplary embodiment, there may beprovided a device for recognizing a motion of an object, the deviceincluding a vision sensor configured to sense the motion and generate atleast one event signal based on the sensed motion, and a processorconfigured to store, in an event map, first time information indicatinga time at which intensity of light corresponding to the at least eventsignal is generated, generate an image based on second time informationcorresponding to a predetermined time range among the first timeinformation, and recognize the motion of the object based on the image.

The processor may generate a first image including time informationcorresponding to a first time range among the first time information andgenerate a second image including time information corresponding to asecond time range among the time information. The second time range maybe different from the first time range.

The processor may determine context information based on the first imageand recognize the motion of the object included in the second imagebased on the context information.

The first time range may be wider than the second time range.

One end of the first time range and one end of the second time range maycorrespond to an identical point in time.

The processor may be configured to recognize the motion of the objectfrom the image based on a neural network.

The event map may include a two-dimensional (2D) map corresponding tothe vision sensor and comprises time information in which most recentlygenerated changes in intensity of light correspond to the event signals.

The event map may include a three-dimensional (3D) map generated byadding a time axis to the 2D map corresponding to the vision sensor andinclude a time information history.

The vision sensor may include an event-based vision sensor configured togenerate at least one event signal in response to an event in whichlight received from the object is asynchronously changed.

According to an aspect of another exemplary embodiment, there isprovided a method of recognizing a motion of an object, the methodincluding: generating a first image of the object corresponding to afirst time period based on an event map including a plurality of mapelements, the event map indicating a position of at least one of theplurality of map elements at which change of light intensity occurs andindicating, a time at which the change of light intensity occurs, inassociation with the position of the at least one of the plurality ofmap elements; obtaining context information from the first image;generating a second image of the object corresponding to a second timeperiod based on the event map; the second time period being subsequentto the first time period and partially overlapped with the first timeperiod; and determining the motion of the object in the second imagebased on the context information.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects of exemplary embodiments will be moreapparent by describing certain exemplary embodiments, with reference tothe accompanying drawings, in which:

FIG. 1 is a diagram illustrating a device for recognizing a motionaccording to example embodiments;

FIG. 2 is a diagram illustrating a process of storing time informationof event signals according to example embodiments;

FIG. 3 is a diagram illustrating an example of storing time informationin an event map according to example embodiments;

FIG. 4 is a diagram illustrating another example of storing timeinformation in an event map according to example embodiments;

FIGS. 5 through 7 are diagrams illustrating examples of generating animage from an event map according to example embodiments; and

FIG. 8 is a flowchart illustrating a method of recognizing a motionaccording to example embodiments.

DETAILED DESCRIPTION

Hereinafter, exemplary embodiments will be described in detail withreference to the accompanying drawings.

In the following description, like drawing reference numerals are usedfor like elements, even in different drawings. Various alterations andmodifications may be made to the exemplary embodiments, some of whichwill be illustrated in detail in the drawings and detailed description.The matters defined in the description, such as detailed constructionand elements, are provided to assist in a comprehensive understanding ofthe exemplary embodiments. However, it should be understood that theseembodiments are not construed as limited to the illustrated forms andinclude all changes, equivalents or alternatives within the idea and thetechnical scope of this disclosure. It is apparent that the exemplaryembodiments can be practiced without those specifically defined matters.Also, well-known functions or constructions are not described in detailsince they would obscure the description with unnecessary detail.

FIG. 1 is a diagram illustrating a device for recognizing a motionaccording to example embodiments.

Referring to FIG. 1, a device for recognizing a motion 100, hereinafterreferred to as a motion recognizing device 100, includes a vision sensor110 and a processor 120. For example, the motion recognizing device 100may be widely used in an area of image recognition, body motionrecognition, security image monitoring, or medical image analyzing. Themotion recognizing device 100 may be included in various systems and/ordevices, for example, smart phones, tablet computers, laptop computers,desktop computers, televisions, wearable devices, security systems, andsmart home systems.

The vision sensor 110 refers to a device that may generate at least oneevent signal by sensing at least a portion of an object in which amotion occurs and includes, for example, an event-based vision sensorsuch as a dynamic vision sensor (DVS). The vision sensor 110 includes aplurality of sensing elements (e.g., pixels). A sensing element mayoutput an event signal by sensing a generation of a predetermined event.For example, the vision sensor 110 may include an array of pixels. Eachpixel may sense local light and generate an asynchronous address eventwhen light changes by a predetermined relative amount. The address eventmay include an x-axis pixel coordinate, a y-axis coordinate, a sign ofbrightness change, and a time stamp.

For example, when the vision sensor 110 senses an event in whichintensity of light increases in a predetermined sensing element, thepredetermined sensing element may output an ON event signal. Also, whenthe vision sensor 110 senses an event in which the intensity of lightdecreases in the predetermined sensing element, the predeterminedsensing element may output an OFF event signal.

The vision sensor 110 may output an event signal from a sensing elementin which changes in the intensity of light are detected unlike aframe-based vision sensor (e.g., a complementarymetal-oxide-semiconductor (CMOS) image sensor (CIS)) may scan an outputfrom each sensing element on a frame-by-frame basis.

An event of changing the intensity of light received to the visionsensor 110 may be generated based on a motion of an object. For example,in practice, when a light source is set by a lapse of time, and when theobject does not independently emit light, the light received to thevision sensor 110 may be generated from the light source and reflectedby the object. When the object, the light source, and the vision sensor110 are static, the light reflected by the object in a stationary statemay be maintained, in practice. Thus, the intensity of light may notchange and vision sensor 110 may not sense the event of changing a lightintensity. Conversely, when the object is moving, the light reflected bythe moving object may be changed based on the motion of the object.Thus, brightness of the light may change, and the vision sensor 110 maysense the changed brightness. In this example, the motion of the objectmay include a relative motion between the vision sensor 110 and theobject occurring due to the motion of the vision sensor 110 as well as aself-motion of the object.

The event signal output in response to the motion of the object mayinclude information generated in a non-synchronous manner. In thisexample, the information may be similar to an optic nerve signaltransferred from a retina to a brain. For example, the event signal maynot be generated with respect to a stationary object, and may begenerated in response to sensing a moving object.

As an example, when an intensity of light increases by at least apredetermined amount in a third sensing element, the vision sensor 110may output a bit indicating an ON event signal and an address of thethird sensing element. As another example, when an intensity of lightdecreases by at least a predetermined amount in a fourth sensingelement, the vision sensor 110 may output a bit indicating an OFF eventsignal and an address of the fourth sensing element. In this example, anaddress for each sensing element may be expressed by, for example, anumber, a position, and an index.

In an example, the vision sensor 110 may not output information of timeat which the changes in the intensity of light are generated. In thisexample, the processor 120 that processes a signal output from thevision sensor 110 may set a point in time at which an event signal isreceived from the vision sensor 110, as the time at which the changes inthe intensity of light are generated. In another example, the visionsensor 110 may output information of the time at which the changes inthe intensity of light are generated. In this example, the vision sensor110 may include a timer. The processor 120 may also receive theinformation of the time at which the changes in the intensity of lightare generated. Hereinafter, for brevity and conciseness, descriptionswill be provided based on an example in which the vision sensor 110 setsthe point in time at which the processor 120 receives the event signal,as the time at which the changes in the intensity of light aregenerated, in lieu of outputting the time at which the changes in theintensity of light are generated.

Since the vision sensor 110 uses the address of the sensing element fromwhich the event of changing the intensity of light is sensed oraddress/time information on the sensing element, a quantity of processedinformation may be reduced when compared to the frame-based visionsensor. For example, a reaction velocity of an event-based image sensormay be based on a unit less than or equal to a microsecond (μs).

The processor 120 receives event signals from the vision sensor 110 andstores, in an event map, time information in which the changes inintensity of light corresponding to the received event signal aregenerated. The processor 120 may store, in an event map, timeinformation in which changes in intensity of light corresponding to areceived predetermined event signal, and store the time information in amap element corresponding to the received predetermined event signalamong a plurality of map elements included in the event map. In thisexample, the plurality of map elements included in the event map maycorrespond to a plurality of sensing elements included in the visionsensor 110, respectively.

In an example, the processor 120 may store, in the event map, timeinformation in which most recently generated changes in intensity oflight correspond to the event signals. In this example, the event mapmay include a two-dimensional (2D) map corresponding to the visionsensor 110. Accordingly, each of the plurality of map elements includedin the event map may include time information in which most recentlygenerated changes in intensity of light correspond to an event signal ofa relevant map element.

In another example, the processor 120 includes a time informationhistory indicating the changes in intensity of light corresponding tothe event signals. In this example, the event map may include a 3D mapgenerated by adding a time axis to the 2D map corresponding to thevision sensor 110. Therefore, when event signals corresponding to anidentical map element are received at different times, the event map mayinclude the time information on most recently received event signalsamong the event signals, as well as time information on previouslyreceived event signals.

The event map may include the time information indicating the changes inintensity of light corresponding to the event signals. For example, thetime information indicating the changes in intensity of light maycorrespond to information of time at which the event signals arereceived from the vision sensor 110 or information of time at which theevent signals are generated in the vision sensor 110. When the timeinformation corresponds to the information of the time at which theevent signals are generated in the vision sensor 110, a timer thatoutputs the time information may be included in the vision sensor 110.

The processor 120 generates an image based on time informationcorresponding to a predetermined time range among the time informationstored in the event map. The processor 120 may extract time informationincluded in the predetermined time range among the time informationstored in the event map. The processor 120 may generate an imageincluding the extracted time information.

For example, the time range may be set as a range from a predeterminedpoint in time in the past to a present point in time. Also, the timerange may be set as a range from a first predetermined point in time toa second predetermined point in time. In this example, the firstpredetermined point in time and the second predetermined point in timemay be points in time prior to a present point in time, or the firstpredetermined point in time may be a point in time prior to the secondpredetermined point in time. Alternatively, the time range may bevariously changed based on a design. Hereinafter, for brevity andconciseness, descriptions will be provided based on an example in whichthe time range is set as a range from a predetermined point in time inthe past to a present point in time.

The processor 120 generates a plurality of images including timeinformation corresponding to different time ranges. The processor 120generates a first image including time information corresponding to afirst time range among the time information stored in the event map andgenerates a second image including time information corresponding to asecond time range among the time information stored in the event map. Inthis example, the first time range is wider than the second time rangeand one end of the first time range and one end of the second time rangecorrespond to an identical point in time. A number of images generatedfrom the event map may be greater than or equal to three and the numberof images is not limited to the aforementioned descriptions.

Examples in which the processor 120 stores time information in an eventmap and generates an image from the event map will be described indetail with reference to FIGS. 2 through 7.

The processor 120 recognizes the motion of the object based on theimage. The processor 120 recognizes the motion of the object from theimage based on a neural network. In this example, the neural networkrefers to a recognition model implemented by a software or a hardware toimitate a calculation ability of a biological system using a greatnumber of artificial neurons connected by connection lines.

For example, the processor 120 may recognize the motion of the objectfrom the image based on an artificial neural network such as aconvolutional neural network (CNN). Alternatively, the processor 120 mayrecognize the motion of the object from the image based on variousmachine learning schemes such as a support vector machine (SVM) and amultiplayer perception (MLP).

When a plurality of images, for example, the first image and the secondimage, including time information corresponding to the different timeranges, for example, the first time range and the second time range, aregenerated, the processor 120 may recognize the motion of the object fromthe first image and the second image based on the neural network. Theprocessor 120 determines context information for the motion of theobject from the first image including time information included in thefirst time range. In this example, the first time range is wider thanthe second time range. The processor 120 may determine the contextinformation for the motion of the object based on the first imageincluding time information in a wide time range.

The processor 120 recognizes the motion of the object included in thesecond image based on the context information for the motion of theobject. The processor 120 may recognize the motion of the objectoccurring at a point in time at which the motion of the object is to berecognized using the second image including time information in arelatively narrow time range.

In an example, the processor 120 may recognize the motion of the useraccording to the context information determined based on the first imageeven when a motion of which the user makes a V shape with her/his fingeris recognized based on the second image. When a rock-paper-scissors gameplayed by the user is determined to be the context information based onthe first image, the processor 120 may determine the finger in V shaperecognized based on the second image, as scissors. Alternatively, whenthe user posing for a photograph is determined to be the contextinformation based on the first image, the processor 120 may determinethe finger in a V shape recognized based on the second image, as acommand to a camera.

In another example, the processor 120 may recognize the motion of theuser according to the context information determined based on the secondimage even when a motion of the user swinging an arm from a right sideto a left side is recognized based on the second image. When the userplaying tennis is determined to be the context information based on thefirst image, the processor 120 may determine the motion recognized basedon the second image, as a motion of swinging a tennis racket.Alternatively, when the user playing a video game is determined to bethe context information based on the first image, the processor 120 maydetermine the motion recognized based on the second image, as an inputcommander to the video game.

In still another example, the processor 120 may determine an emergencysituation to be the context information based on motions of which peopleare rapidly moving in an identical direction included in the firstimage. In this example, the processor 120 may recognize, based on thesecond image, a motion of which a person among the people is falling.The processor 120 may recognize a person who has fallen as a person inneed of assistance.

FIG. 2 is a diagram illustrating a process of storing time informationof event signals according to example embodiments.

Referring to FIG. 2, when event signals are received, the processor 120illustrated in FIG. 1 stores, in an event map 200, information of timeat which intensity of light corresponding to the event signals changes.The event map 200 may include a plurality of map elements to store timeinformation on a plurality of event signals. Each of the map elementsmay store time information on an event signal corresponding to arelevant map element. For example, time information T_(i, j) mayindicate a change in intensity of light corresponding to an event signalin a map element 210 at a position (i, j), and time informationT_(i+1,j) may indicate a change in intensity of light corresponding toan event signal in a map element 220 at a position (i+1, j).

When an event signal is received, the processor 120 updates a mapelement corresponding to a relevant event signal. The processor 120 mayupdate a map element corresponding to the received signal, in lieu ofupdating all map elements. For example, the processor 120 may detect themap element corresponding to the received event signal among theplurality of map elements included in the event map 200 and update thedetected map element as time information on the received event signal.

In this example, time information in which changes in intensity of lightcorresponding to event signals are generated may be time information inwhich the event signals are received from a vision sensor or timeinformation in which the event signals are generated in the visionsensor. For example, the time information in which the changes inintensity of light corresponding to the event signals are generatedrefers to a time stamp.

FIG. 3 is a diagram illustrating an example of storing time informationin an event map according to example embodiments.

In FIG. 3, a motion 310 of an object that moves from right to left isillustrated.

In an example, an event map 320 includes a time at which changes inintensity of light corresponding to received event information aregenerated, as time information and includes map elements in a 5×5arrangement. For example, map elements at a position (1, n) included inthe event map include “1”, map elements at a position (2, n) include“2”, and map elements at a position (5, n) include “5”. In this example,n indicates a constant between 1 through 5.

In another example, an event map 330 includes a result to which apredetermined conversion function is applied at a time at which thechanges in intensity of light corresponding to the received eventinformation are generated. For example, a conversion function includes(i) a function to subtract a present time from a time at which changesin intensity of light are generated, (ii) a function to subtract thetime at which the changes in intensity of light are generated from thepresent time, (iii) a function to add a greater weight as the time atwhich the changes in an intensity of light are generated becomes closerto the present time, and (iv) a function to convert the time at whichthe changes in intensity of light are generated into a pixel value ingrayscale.

The event map 330 illustrated in FIG. 3 converts the time at which thechanges in intensity of light corresponding to the received eventinformation are generated into a pixel value in grayscale and includesthe converted pixel value. For example, the event map 330 may representthe time at which the changes in intensity of light are generated asgrayscale by converting the time included in the event map 320 intoconstants within a grayscale range between 0 (black) and 255 (white). Aschanges in intensity of light are most recently generated, the event map330 may represent time information in which the most recently generatedchanges in intensity of light to be closer to black.

In still another example, an event map 340 includes map elements in a7×7 arrangement. Based on the event map 340 including relatively moremap elements, the motion of the object may be represented in a moredetailed manner.

In FIG. 3, the event frames 320 through 340 including the map elementsin sizes of 5×5 or 7×7 are only examples and not limited thereto. Eventframes may include map elements in a predetermined number, for example,128×128.

FIG. 4 is a diagram illustrating another example of storing timeinformation in an event map according to example embodiments.

In FIG. 4, a motion 410 of an object that moves from a lower left sideto an upper right side in zigzag is illustrated. In the motion 410, acircle indicates a start point of the motion 410, and a cross mark(i.e., x) indicates an end point of the motion 410.

In an example, an event map 420 includes a 3D map by adding a time axisto a 2D map corresponding to a vision sensor. The event map 420 mayinclude time information history in which changes in intensity of lightcorresponding to event signals are generated by adding the time axis tothe 2D map including map elements provided in an x-axis and a y-axis.Events are time-stamped with microsecond resolution and transmittedasynchronously at the time they occur. Each event may be represented as(x_(k), y_(k), t_(k), p_(k)), where x_(k) and y_(k) are pixelcoordinates of the event and t_(k) is its time-stamp. The parameterp_(k) has a value of +1 or −1 and indicates the change of brightness(i.e., polarity).

The event map 420 includes time information in which most recentlygenerated changes in intensity of light corresponds to event signals inaddition to time information in which changes in intensity of light arepreviously generated, even when the event signals corresponding to anidentical map element are received at different times. The event map 420includes time at which the changes in intensity of light are generatedor a pixel value in grayscale converted from the time at which thechanges in intensity of light are generated. The event map 420 mayrepresent time at which the changes in intensity of light are generatedas grayscale. The event map 420 may represent time information in whichthe most recently generated changes in intensity of light to be closerto black. When the range of intensity values is set from 0 (black) to255 (white), the more recent the time information is generated, thecloser the intensity value is to 0. The time information represented inthe event map 420 may be positioned to be adjacent to each other withouta discontinuity in an X-axis direction.

Hereinafter, for conciseness, descriptions will be provided based on anexample in which an event map includes a 2D map.

FIGS. 5 through 7 are diagrams illustrating examples of generating animage from an event map according to example embodiments.

Referring to FIG. 5, time information 510 stored in an event map 500 isillustrated. For example, as illustrated in FIG. 5, time information ofan event signal corresponding to a map element at a position (1, 2) is“1”, and time information of an event signal corresponding to a mapelement at a position (3, 3) is “3”.

A map element not including the time information among map elementsincluded in the event map 500 includes a null value, and the map elementmay indicate that an event signal corresponding to the map element isnot received from a vision sensor.

Referring to FIG. 6, a first image 600 generated from the event map 500is illustrated. The first image 600 includes time information 610corresponding to a first time range and the first time range may be setas a range from “3” to “5”.

The first image 600 includes the time information 610 extracted to beincluded in the first time range among the time information 510 includedin the event map 500. Time information not included in the first timerange among the time information 510 included in the event map 500 maynot be included in the first image 600.

Referring to FIG. 7, a second image 700 generated from the event map 500is illustrated. The second image 700 includes time information 710corresponding to a second time range, and the second time range may beset as a range corresponding to “5”.

The second image 700 includes the time information 710 extracted to beincluded in the second time range among the time information 510included in the event map 500. Time information not included in thesecond time range among the time information 510 included in the eventmap 500 may not be included in the second image 700.

The processor 120 illustrated in FIG. 1 may determine, based on thefirst image 600, context information in which an object moves from acenter to a lower right side, and recognize, based on the second image700, an accurate position of the object moving toward a lower rightside. The motion of the object is illustrated in FIGS. 6 and 7 for easeof description, a motion recognized based on a plurality of images isnot limited thereto. The motion of the object may be recognized based onvarious forms of context information.

FIG. 8 is a flowchart illustrating a method of recognizing a motionaccording to example embodiments.

A method of recognizing a motion may be performed by a processor 120included in a motion recognizing device 100.

In operation 810, the processor 120 may receive event signals from avision sensor 110 to generate at least one event signal by sensing atleast a portion of the object in which the motion occurs.

The vision sensor 110 refers to a device that may generate at least oneevent signal in response to an event in which light received from theobject is asynchronously changed. For example, the vision sensor 110 mayinclude a dynamic vision sensor (DVS). The vision sensor 110 maytransmit only the local pixel-level changes caused by movement in ascene, at the time the changes occur, instead of transmitting entireimages at fixed frame rates.

In operation 820, the processor 120 may store, in an event map, timeinformation in which changes in intensity of light corresponding to theevent signals are generated. In an example, the event map includes a 2Dmap corresponding to the vision sensor 110, and includes the timeinformation in which most recently generated changes in intensity oflight correspond to the event signals. In another example, the event mapincludes a 3D map by adding a time axis to the 2D map corresponding tothe vision sensor 110 and includes a time information history in whichthe changes in intensity of light corresponding to the event signals aregenerated.

In this example, the time information in which the changes in intensityof light are generated may be time information in which the eventsignals are received from the vision sensor or time information in whichthe event signals are generated in the vision sensor.

In operation 830, the processor 120 may generate an image based on timeinformation corresponding to a predetermined time range among the timeinformation stored in the event map.

In this example, the time range may be specified based on differentpoints in time. Alternatively, the time range may be set as a range froma first predetermined point in time to a second predetermined point intime. Here, the first predetermined point in time and the secondpredetermined point in time may be points in time prior to a presentpoint in time, or the first predetermined point in time may be a pointin time prior to the second predetermined point in time. Alternatively,the time range may be variously changed based on a design.

The processor 120 may generate a plurality of images including timeinformation corresponding to different time ranges. For example, theprocessor 120 may generate a first image including time informationcorresponding to a first time range among the time information stored inthe event map. The processor 120 may generate a second image includingtime information corresponding to a second time range among the timeinformation stored in the event map. In this example, the first timerange is wider than the second time range, and one end of the first timerange and one end of the second time range correspond to an identicalpoint in time.

In operation 840, the processor 120 may recognize a motion of an objectbased on the image. For example, the processor 120 may recognize themotion of the object from the image based on a neural network.Alternatively, the processor 120 may recognize the motion of the objectfrom the image based on various machine learning schemes such as a CNNand an SVM.

In operation 830, when a plurality of images including time informationcorresponding to the different time ranges is generated, the processor120 may recognize the motion of the object based on the plurality ofimages. For example, when (i) the first image corresponds to the firsttime range and (ii) the second image corresponds to the second timerange having a narrower time range than the first time range aregenerated, the processor 120 may determine context information for themotion of the object from the first image. The processor 120 mayrecognize the motion of the object included in the second image based onthe context information. The processor 120 may recognize the motion ofthe object occurring at a point in time at which the motion of theobject is to be recognized using the second image including timeinformation in a relatively narrow time range.

Repeated descriptions will be omitted for brevity and concisenessbecause the descriptions provided with reference to FIGS. 1 through 7are also applicable to operations illustrated in FIG. 8.

According to an aspect of example embodiments, it is possible toefficiently reduce an amount of operation by recognizing a motion of anobject based on an image including time information in which changes inintensity of light corresponding to received event information aregenerated.

According to another aspect of example embodiments, it is possible torepresent a motion of an object represented based on a plurality offrame-based images including a brightness value of the object in a lessnumber of images by representing the motion of the object based on animage including time information in which changes in intensity of lightcorresponding to received event information are generated and

According to still another aspect of example embodiments, it is possibleto efficiently enhance a recognition speed with respect to a motion ofan object by recognizing the motion of the object based on an imageincluding time information on an event signal in lieu of using an imageincluding a brightness value of the object.

According to a further aspect of example embodiments, it is possible toefficiently enhance recognition accuracy with respect to a motion of anobject by recognizing the motion based on context information for themotion of the object based on a plurality of images including timeinformation corresponding to different time ranges.

The exemplary embodiments described herein may be implemented usinghardware components and software components. For example, the hardwarecomponents may include microphones, amplifiers, band-pass filters, audioto digital convertors, and processing devices. A processing device maybe implemented using one or more general-purpose or special purposecomputers, such as, for example, a processor, a controller and anarithmetic logic unit, a digital signal processor, a microcomputer, afield programmable array, a programmable logic unit, a microprocessor orany other device capable of responding to and executing instructions ina defined manner. The processing device may run an operating system (OS)and one or more software applications that run on the OS. The processingdevice also may access, store, manipulate, process, and create data inresponse to execution of the software. For purpose of simplicity, thedescription of a processing device is used as singular; however, oneskilled in the art will appreciated that a processing device may includemultiple processing elements and multiple types of processing elements.For example, a processing device may include multiple processors or aprocessor and a controller. In addition, different processingconfigurations are possible, such a parallel processors.

The software components may include a computer program, a piece of code,an instruction, or some combination thereof, to independently orcollectively instruct or configure the processing device to operate asdesired. Software and data may be embodied permanently or temporarily inany type of machine, component, physical or virtual equipment, computerstorage medium or device, or in a propagated signal wave capable ofproviding instructions or data to or being interpreted by the processingdevice. The software also may be distributed over network coupledcomputer systems so that the software is stored and executed in adistributed fashion. The software and data may be stored by one or morenon-transitory computer readable recording mediums.

The above-described exemplary embodiments may be recorded innon-transitory computer-readable media including program instructions toimplement various operations which may be performed by a computer. Themedia may also include, alone or in combination with the programinstructions, data files, data structures, and the like. The programinstructions recorded on the media may be those specially designed andconstructed for the purposes of the exemplary embodiments, or they maybe of the well-known kind and available to those having skill in thecomputer software arts. Examples of non-transitory computer-readablemedia include magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD ROM discs and DVDs;magneto-optical media such as optical discs; and hardware devices thatare specially configured to store and perform program instructions, suchas read-only memory (ROM), random access memory (RAM), flash memory, andthe like. The media may be transfer media such as optical lines, metallines, or waveguides including a carrier wave for transmitting a signaldesignating the program command and the data construction. Examples ofprogram instructions include both machine code, such as code produced bya compiler, and files containing higher level code that may be executedby the computer using an interpreter. The described hardware devices maybe configured to act as one or more software modules in order to performthe operations of the above-described exemplary embodiments, or viceversa.

The foregoing exemplary embodiments and advantages are merely exemplaryand are not to be construed as limiting. The present teaching can bereadily applied to other types of apparatuses. Also, the description ofthe exemplary embodiments is intended to be illustrative, and not tolimit the scope of the claims, and many alternatives, modifications, andvariations will be apparent to those skilled in the art.

What is claimed is:
 1. A method of recognizing a motion of an objectusing a dynamic vision sensor configured to detect intensity of light bya movement of the object, the method comprising: generating eventsignals from the dynamic vision sensor comprising a plurality of sensingelements; storing, in an event map comprising a plurality of mapelements corresponding to the plurality of sensing elements, first timeinformation indicating a time at which intensity of light correspondingto the event signals changes; generating a first map image based onsecond time information corresponding to a first time range among thefirst time information; generating a second map image based on thirdtime information corresponding to a second time range among the firsttime information; recognizing the motion of the object based on thefirst map image and the second map image, and wherein the first timerange comprises at least a part of the second time range.
 2. The methodof claim 1, wherein the recognizing the motion comprises: determiningcontext information based on the first map image; and recognizing themotion of the object corresponding to the second map image based on thecontext information.
 3. The method of claim 1, wherein the first timerange is wider than the second time range.
 4. The method of claim 1,wherein one end of the first time range and one end of the second timerange correspond to an identical point in time.
 5. The method of claim1, wherein the recognizing the motion of the object comprisesrecognizing the motion of the object from the first map image and thesecond map image based on a neural network.
 6. The method of claim 1,wherein the first time information indicates most recently time at whicheach of the plurality of sensing elements generates at least one of theevent signals.
 7. The method of claim 1, wherein the first timeinformation indicates history in which each of the plurality of sensingelements generates at least one of the event signals.
 8. The method ofclaim 1, wherein the dynamic vision sensor is configured to generate theevent signals in response to events in which light is asynchronouslychanged based on the motion of the object.
 9. A device for recognizing amotion of an object, the device comprising: a dynamic vision sensorcomprising a plurality of sensing elements configured to generate atleast one event signal in response to change in intensity of light basedon the motion of the object; and a processor configured to: store, in anevent map, first time information indicating a time at which intensityof light corresponding to the at least one event signal is changed;generate a first map image based on second time informationcorresponding to a first time range among the first time information;generate a second map image based on third time informationcorresponding to a second time range among the first time information;recognize the motion of the object based on the first map image and thesecond map image, wherein the first time range comprises at least a partof the second time range.
 10. The device of claim 9, wherein theprocessor is further configured to: determine context information basedon the first map image; and recognize the motion of the objectcorresponding to the second map image based on the context information.11. The device of claim 9, wherein the first time range is wider thanthe second time range.
 12. The device of claim 9, wherein one end of thefirst time range and one end of the second time range correspond to anidentical point in time.
 13. The device of claim 9, wherein the firsttime information comprises at least one time value at which at least oneof the plurality of sensing elements generates the at least one eventsignal.
 14. The device of claim 9, wherein the first time informationcomprises at least one result to which a conversion function is appliedat a time at which the change in intensity of light corresponding to theat least one event signal.
 15. The device of claim 9, wherein the firsttime information comprises at least one pixel value in grayscalegenerated based on at least one time value at which at least one of theplurality of sensing elements generates the at least one event signal.16. The device of claim 9, wherein the first time information comprisesmost recently time at which at least one of the plurality of sensingelements generates the at least one event signal.
 17. The device ofclaim 9, wherein the first time information indicates history in whichat least one of the plurality of sensing elements generates the at leastone event signal.
 18. A device for recognizing a motion of an object,the device comprising: a dynamic vision sensor comprising a plurality ofsensing elements configured to generate event signals in response tochange in intensity of light based on the motion of the object; and aprocessor configured to: store, in an event map, first time informationindicating a time at which intensity of light corresponding to the eventsignals are changed; generate a map image based on second timeinformation corresponding to a predetermined time range among the firsttime information, wherein the map image comprises time values thatsatisfy the predetermined time range, each of the time values beingmapped to a respective position in the map image; and recognize themotion of the object based on the map image, wherein the event mapcomprises a two-dimensional (2D) map corresponding to the dynamic visionsensor, the 2D map comprising a plurality of map elements correspondingto the plurality of sensing elements, and the event map indicateshistory in which the plurality of sensing elements generates the eventsignals.
 19. The device of claim 18, wherein the processor is configuredto generate a first map image comprising time information correspondingto a first time range among the first time information and generate asecond map image comprising time information corresponding to a secondtime range among the time information, the first time range comprises atleast a part of the second time range.
 20. The device of claim 19,wherein the processor is configured to determine context informationbased on the first map image and recognize the motion of the objectcorresponding to the second map image based on the context information.